Method and device of image processing and camera

ABSTRACT

An image processing method includes obtaining an acquired image captured by an image acquisition device and obtaining vibration information associated with the acquired image and generated while the image acquisition device capturing the acquired image. The vibration information includes an angle of vibration. The method further includes performing a distortion correction to the acquired image to obtain a distortion-corrected image based upon the vibration information and a preset distortion correction parameter, and determining a target image from the distortion-corrected image based upon the vibration information.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of InternationalApplication No. PCT/CN2015/085641, filed on Jul. 31, 2015, the entirecontents of which are incorporated herein by reference.

TECHNICAL FIELD

The disclosure relates to image processing technology, and moreparticularly to a method and device of image processing and a camera.

BACKGROUND OF THE DISCLOSURE

Cameras convert optical image signals into electrical signals forsubsequent transmission and storage. A lens is a key component of acamera. Lenses can include a standard lens and a wide-angle lens interms of angle of view. An angle of view of a wide-angle lens can bemore than 90 degrees. A fisheye lens can even have an angle of view ofapproximately 180 degrees, making it possible to capture a wide angularextent of a scene.

However, jello effect and ripple-like effect can occur in capturedimages if the camera is subjected to vibrations in capturing images,which severely affect a quality of the images.

In prior art, an anti-vibration can be effected by a mechanicalstructure. For instance, a physical gimbal can be used where ananti-vibration is effected with a three-axis motion compensation.However, a physical gimbal can be costly and inconvenient to use (e.g.,a user has to carry it as an extra load).

SUMMARY OF THE DISCLOSURE

The disclosure provides a method and device of image processing and acamera. With the technical solution of the disclosure, a satisfactoryanti-vibration can be effected, and a high-quality image can beobtained.

A first aspect of the disclosure provides a method of image processing.The method can comprise obtaining an acquired image captured by an imageacquisition device; obtaining a vibration information associated withthe acquired image, which vibration information being generated as theimage acquisition device capturing the acquired image, the vibrationinformation comprising an angle of vibration; and determining a targetimage from the acquired image based upon the vibration information.

A second aspect of the disclosure provides a method of image processing.The method can comprise obtaining an acquired image captured by an imageacquisition device; obtaining a vibration information associated withthe acquired image, which vibration information being generated as theimage acquisition device capturing the acquired image, the vibrationinformation comprising an angle of vibration; performing a distortioncorrection to the acquired image based upon the vibration informationand a preset distortion correction parameter; and determining a targetimage from the distortion-corrected image based upon the vibrationinformation.

A third aspect of the disclosure provides device of image processing.The device can comprise an image obtaining module configured to obtainan acquired image captured by an image acquisition device; a vibrationobtaining module configured to obtain a vibration information associatedwith the acquired image, which vibration information being generated asthe image acquisition device capturing the acquired image, the vibrationinformation comprising an angle of vibration; and a processing moduleconfigured to determine a target image from the acquired image basedupon the vibration information.

A fourth aspect of the disclosure provides a camera as an example imageacquisition device. The camera can comprise a lens; an image sensorconfigured to acquire an image data through the lens; and an imageprocessor connected to the image sensor. The image processor can beconfigured to obtain a vibration information associated with theacquired image, which vibration information being generated as the imageacquisition device capturing the acquired image, the vibrationinformation comprising an angle of vibration, and determine a targetimage from the acquired image based upon the vibration information. Thevibration information can comprises an angle of vibration.

A fifth aspect of the disclosure provides a device of image processing.The device can comprise a first obtaining module configured to obtain anacquired image captured by an image acquisition device; a secondobtaining module configured to obtain a vibration information associatedwith the acquired image, which vibration information being generated asthe image acquisition device capturing the acquired image, the vibrationinformation comprising an angle of vibration; a first processing moduleconfigured to perform a distortion correction to the acquired imagebased upon the vibration information and a preset distortion correctionparameter; and a second processing module configured to determine atarget image from the distortion-corrected image based upon thevibration information.

A sixth aspect of the disclosure provides a camera as an example imageacquisition device. The camera can comprise a lens; an image sensorconfigured to acquire an image data through the lens; and an imageprocessor connected to the image sensor. The image processor can beconfigured to obtain an acquired image from the image data acquired bythe image sensor; obtain a vibration information associated with theacquired image, which vibration information being generated as the imageacquisition device capturing the acquired image, the vibrationinformation comprising an angle of vibration; perform a distortioncorrection to the acquired image based upon the vibration informationand a preset distortion correction parameter; and determine a targetimage from the distortion-corrected image based upon the vibrationinformation.

A seventh aspect of the disclosure provides an aerial vehicle for aerialphotography. The aerial vehicle can comprise a body and a camera mountedon the body. The camera can comprise a camera of the fourth aspect or acamera of the sixth aspect.

A eighth aspect of the disclosure provides an aerial vehicle for aerialphotography. The aerial vehicle can comprise a body, an imageacquisition device mounted on the body and a processor in datacommunication with the image acquisition device. The processor cancomprises a device of image processing of the third aspect or a deviceof image processing of the fifth aspect.

With embodiments of the disclosure, an image processing can be performedbased upon a vibration information. An anti-vibration can be effected inthe process of image processing without using an anti-vibration hardwaredevice (e.g., a gimbal). A cost can be reduced, and a user experiencecan be improved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a flowchart of a method of image processing in accordancewith embodiments of the disclosure;

FIG. 2 shows a flowchart of a method of image processing in accordancewith another embodiment of the disclosure;

FIG. 3 shows a flowchart of a method for determining a coordinatemapping in accordance with embodiments of the disclosure;

FIG. 4 shows a mapping of a point in a target image coordinate systeminto a world coordinate system in accordance with embodiments of thedisclosure;

FIG. 5 shows a rotation from a world coordinate system to a tangentplane coordinate system;

FIG. 6 shows a rotation from a tangent plane coordinate system to atarget image coordinate system;

FIG. 7 shows a distortion correction in accordance with embodiments ofthe disclosure;

FIG. 8 shows a flowchart of a method of image processing in accordancewith another embodiment of the disclosure;

FIG. 9 shows a between a distortion-corrected image and a target imagein accordance with embodiments of the disclosure;

FIG. 10 shows a configuration of a device of image processing inaccordance with embodiments of the disclosure;

FIG. 11 shows a configuration of a processing module in the device ofimage processing of FIG. 10;

FIG. 12 shows a configuration of a camera in accordance with embodimentsof the disclosure;

FIG. 13 shows a configuration of a device of image processing inaccordance with another embodiment of the disclosure;

FIG. 14 shows a configuration of a first processing module in the deviceof image processing of FIG. 13; and

FIG. 15 shows a configuration of a camera in accordance with embodimentsof the disclosure.

DETAILED DESCRIPTION OF THE DISCLOSURE

A better understanding of the disclosure will be obtained by referenceto the following detailed description that sets forth illustrativeembodiments with reference to the drawings. It will be apparent that,the embodiments described herein are merely provided by way of example.Those skilled in the art can conceive various embodiments in light ofthose disclosed herein without inventive efforts, and all theseembodiments are within the scope of the disclosure.

FIG. 1 shows a flowchart of a method of image processing in accordancewith embodiments of the disclosure. The method of image processing inaccordance with embodiments of the disclosure can be applied to variousimage acquisition devices and can be implemented by one or more imageprocessors. In some embodiments, the method can comprise steps S101 toS103.

In step S101, an acquired image captured by an image acquisition devicecan be obtained.

The image acquisition device can include an intelligent image capturingdevice capable of capturing images, such as a still camera or a videocamera. An acquired image can be obtained from the image acquisitiondevice through a data communication therewith. In some instances, theimage acquisition device can comprise a wide-angle lens such as afisheye lens, and the acquired image can be a fisheye image.

In step S102, a vibration information associated with the acquiredimage, which vibration information is generated as the image acquisitiondevice capturing the acquired image, can be obtained. In some instances,the vibration information can comprise an angle of vibration.

In some embodiments, the image acquisition device can operate in a videocapture mode or in a continuous capture mode. The vibration informationcan be an angular change in a position of the image acquisition devicein capturing a current image relative to a position of the imageacquisition device in capturing a previous image. Alternatively, thevibration information can be an angular change in a position of theimage acquisition device in capturing a current image relative to areference position. The reference position can be an initial position ofthe image acquisition device where it starts to capture images. In someinstances, the vibration information can include angular changeinformation of the image acquisition device in three directions. Forinstance, the vibration information can comprise angular changeinformation in a pitch angle, a yaw angle and a roll angle.

In some instances, an angular sensor (e.g., a gyroscope) can be providedat the image acquisition device to detect the vibration information ofthe image acquisition device. A raw data can be obtained directly fromthe angular sensor, and the angular change can be calculated from theraw data. Optionally, the angular change can be obtained directly fromthe angular sensor provided that the angular sensor is provided with acalculation capability.

In some embodiments, the angular sensor can be directly provided on theimage acquisition device through a fixed connection. Alternatively, theangular sensor can be fixedly connected to various external devices towhich the image acquisition device is rigidly connected. For instance,the angular sensor can be mounted on an aerial vehicle which carries theimage acquisition device. The angular sensor fixed connected to theexternal device can detect the vibration information of the imageacquisition device as a vibration of the external devices directly leadsto a vibration of the image acquisition device. It will be appreciatedthat, the angular sensor may not accurately detect the vibrationinformation of the image acquisition device if the image acquisitiondevice is flexibly connected to the external device. In this situation,the angular sensor may not be provided on the external device.

In step S103, a target image can be determined from the acquired imagebased upon the vibration information.

In some instances, the process in step S103 can comprise performing,based upon the vibration information, a coordinate transformation toobtain a coordinate mapping from a plane coordinate system in which theacquired image is positioned and a target coordinate system, andsubsequently determining a texture information of respective pixel(e.g., a grayscale value corresponding to the pixel) in the acquiredimage to obtain a texture mapping. A conversion from the acquired imageto a target image can be performed based upon the obtained coordinatemapping, the texture mapping and a preset distortion correctionparameter, such that a distortion-corrected target image can beobtained.

In some instances, the process in step S103 can comprise performing,based upon the vibration information, a coordinate transformation toobtain a coordinate mapping from a plane coordinate system in which theacquired image is positioned and a target coordinate system, andsubsequently determining a texture information of respective pixel(e.g., a grayscale value corresponding to the pixel) in the acquiredimage to obtain a texture mapping. A conversion from the acquired imageto an initial image can be performed based upon the obtained coordinatemapping, the texture mapping and a preset distortion correctionparameter, such that a distortion-corrected image can be obtained.Subsequently, a vibration compensation can be performed based upon thevibration information and a preset angle of view, such that adistortion-corrected target image can be obtained from thedistortion-corrected image.

In some instances, the vibration compensation can comprise determiningan initial region in the distortion-corrected image based upon thepreset angle of view, and performing a compensation to the initialregion based upon the vibration information to obtain a target region.An image covered by the target region can be the target image. In someembodiments of the disclosure, the preset angle of view can be smallerthan an angle of view of the image acquisition device capturing theacquired image. In some instances, the preset angle of view can be anangle of view of a standard image, while the acquired image can have anangle of view of a fisheye image. Optionally, the acquired image canhave a standard angle of view, while the preset angle of view can besmaller than the standard angle of view.

For instance, in performing the compensation, if a pitch angleinformation in the vibration information indicates that the imageacquisition device rotates downwardly by a first angle, then the initialregion can be moved upwardly by the first angle. For instance, inperforming the compensation, if a yaw angle information in the vibrationinformation indicates that the image acquisition device rotates to theleft by a second angle, then the initial region can be moved to theright by the second angle. In this way, the vibration compensation canbe effected.

In embodiments of the disclosure, an image processing can be performedbased upon a vibration information. An anti-vibration can be effected inthe process of image processing without using an anti-vibration hardwaredevice (e.g., a gimbal). A cost can be reduced, and a user experiencecan be improved.

FIG. 2 shows a flowchart of a method of image processing in accordancewith another embodiment of the disclosure. The method of imageprocessing in accordance with embodiments of the disclosure can beapplied to various image acquisition devices, and can be implemented byone or more image processors. In some embodiments, the method cancomprise steps S201 to S207.

In step S201, an acquired image captured by an image acquisition devicecan be obtained.

The acquired image can be captured using various cameras. In someembodiments, the acquired image can be a wide-angle image, from which atarget image having an angle of view smaller than that of the wide-angleimage can be determined.

In step S202, a vibration information associated with the acquiredimage, which is generated as the image acquisition device capturing theacquired image, can be obtained. In some instances, the vibrationinformation can comprise an angle of vibration.

In some instances, the vibration information can be an angular change ina position of the image acquisition device in capturing the acquiredimage relative to a position of the image acquisition device incapturing a previously acquired image. Optionally, the vibrationinformation can be an angular change in a position of the imageacquisition device in capturing the acquired image relative to areference position. The vibration information can comprise a pitchangle, a yaw angle and/or a roll angle.

The vibration information can be collected by a sensor (e.g., agyroscope) provided at the image acquisition device. The vibrationinformation can subsequently be used in a plane coordinate mappingprocessing and a vibration compensation to determine a target region.The vibration information can comprise an angle information in threedimensions, as discussed hereinabove. In some instances, the vibrationinformation can be calculated from an angular acceleration measured bythe gyroscope.

In step S203, a synchronization can be performed to the obtainedacquired image and the vibration information.

The synchronization can be performed to align the data measured by thesensor (e.g., the gyroscope) with the acquired image captured by acamera in timeline, such that the vibration information is associatedwith the obtained acquired image.

In some instances, a capture time at which the acquired image beingcaptured can be obtained, and a vibration data measured by the sensor(e.g., the gyroscope) at the capture time can be obtained. The vibrationinformation of a currently acquired image can be obtained based upon avibration data at a capture time the currently acquired image beingcaptured and a vibration data at a time a previously acquired imagebeing captured. Accuracy of the vibration information of the currentlyacquired image may be ensured based upon the capture time of thecurrently acquired image and time points at which the vibration data ofthe currently acquired image and the previously acquired image beingmeasured, thereby reducing or avoiding errors in subsequent processing.

Once the synchronization is successful, a trigger information can beprovided to trigger a process of determining the target image from theacquired image. In some instance, the step S204 can be performed.

In step S204, a mapping from an image coordinate system in which theacquired image is positioned and a target coordinate system can bedetermined based upon the angle of vibration.

In some instances, a coordinate transformation and mapping in step S204can be effected using a world coordinate system which is fixed. Atransformation from the image coordinate system in which the acquiredimage is positioned to the target coordinate system can be effected by atransformation from the image coordinate system to the world coordinatesystem, such that the mapping from the image coordinate system in whichthe acquired image is positioned to the target coordinate system can beindirectly obtained.

In step S205, a preset distortion correction parameter can be obtained.

In step S206, each point in the acquired image can be mapped to acorresponding location in the target coordinate system based upon thedetermined mapping from the image coordinate system in which theacquired image is positioned to the target coordinate system and thedistortion correction parameter.

In step S207, a pixel value of each point in the acquired image can beobtained, and a target image can be generated based upon the obtainedpixel value and the location of each point.

It will be appreciated that, the process of mapping each point in theacquired image to the corresponding location in the target coordinatesystem in step S206 and the process of obtaining the pixel value of eachpoint in the acquired image in step S207 can be performedsimultaneously.

FIG. 3 shows a flowchart of a method for determining a coordinatemapping in accordance with embodiments of the disclosure. In someembodiments, the method can correspond to the step S204 as describedhereinabove and comprise steps S301 to S303.

In step S301, a mapping from the image coordinate system in which theacquired image is positioned to a world coordinate system can beestablished based upon the angle of vibration.

In step S302, a mapping from the target coordinate system to the worldcoordinate system can be determined based upon the angle of vibration.

In step S303, the mapping from the image coordinate system in which theacquired image is positioned to the target coordinate system can bedetermined based upon the world coordinate system.

In some embodiments, the process in step S302 can comprise establishinga mapping from the world coordinate system to a preset tangent planecoordinate system based upon the angle of vibration, establishing amapping from the target coordinate system to the preset tangent planecoordinate system based upon the angle of vibration, and determining themapping from the target coordinate system to the world coordinate systembased upon the preset tangent plane coordinate system. In someinstances, the tangent plane coordinate system can be a tangent planecoordinate system in which an arbitrary point on a hemispherical surfacehaving a center coinciding with a center of a lens of the imageacquisition device is positioned.

The process of performing the coordinate transformation and the processof performing distortion correction to the acquired image (e.g., afisheye image) in embodiments of the disclosure will be described.

In a first aspect, let a coordinate system in which a fisheye image(e.g., a distorted image) acquired by an imaging system be an imagecoordinate system O₂-uv, where O₂ is a central point of the image, andu, v are a horizontal axis and a vertical axis of the image,respectively.

In a second aspect, let an actual world coordinate system be O_(r)-XYZ.For the sake of a general description, with regard to a fisheye lenshaving a certain focal length, we can assume an object space of the lensbeing distributed over a hemispherical surface having a centercoinciding with a center O_(r) of the lens. We can also intuitivelyassume an image formed in an image space O₂-uv being a projection of theobject space onto an image sensor along an optical axis.

In a third aspect, let a camera coordinate system beO_(b)-X_(b)Y_(b)Z_(b). The camera coordinate system can be stationerywith respect to a camera and a sensor (e.g., a gyroscope). A variationparameter of the camera coordinate system can be obtained using thesensor such as the gyroscope in real-time.

In a fourth aspect, let a coordinate system O1-xy in which a targetimage (which can be an image having a standard angle of view) ispositioned be an image coordinate system of a final output image.

In a fifth aspect, let a tangent plane coordinate system in which apoint O on the spherical surface in the second aspect is positioned beO-X′Y′. We can consider the tangent plane coordinate system as an objectcoordinate system corresponding to the image coordinate system in thefourth aspect.

A hardware configuration of the system can comprise the gyroscope, thecamera (e.g., a camera provided with a fisheye lens) and a GPU. Apositional relation between the gyroscope and the camera (e.g., a cameraprovided with a fisheye lens) can be fixed.

In some embodiments, the process of the coordinate transformation andthe process of distortion correction in accordance with embodiments ofthe disclosure can comprise obtaining offline calibration parameters,obtaining input parameters of each frame and outputting adistortion-corrected image in real-time.

In some embodiments, the offline calibration parameters can comprise acoordinate of a center point of the fisheye image, a length and a widthof the fisheye image, a length and a width of the standard image, acorrection parameter f of the focal length, and an angle of view γ. Theinput parameters of each frame can comprise an angular change in aposition of the camera relative to a position of the camera capturing aprevious frame, including a pitch angle θ, a yaw angle φ and a rollangle ρ, a current fisheye image frame, a pitch angular velocity {dotover (θ)}, a yaw angular velocity {dot over (φ)} and a roll angularvelocity {dot over (ρ)}.

In some embodiments, a frame rate of the camera can be set, and thecamera can then be directed to operate and acquire fisheye images. Inthe meanwhile, a parameter representing a positional change of thecamera can be obtained from the gyroscope. The data acquired by thegyroscope and the data acquired by the camera can aligned with eachother in a timeline through a synchronization process. Subsequently, theacquired data and the offline calibration parameters can be input into agraphics processing unit (GPU) to calculate and output the target image.In some embodiments, the calculation process can comprise transforming acoordinate in the target image into a coordinate under the worldcoordinate system through the tangent plane coordinate system, andtransforming the coordinate under the world coordinate system into acoordinate under the fisheye image coordinate system.

As shown in FIG. 4, a coordinate system in which an actual scenecorresponding to an original fisheye image is positioned can be Or-XYZ,where the scene can be positioned on the illustrated hemisphericalsurface. A screen image can be the target image. A point O in a fisheyescene (e.g., the actual scene) corresponding to a central point O₁ ofthe target image can be positioned on the hemispherical surface. A pointP₁ (x, y) in the target image can correspond to a point P in a tangentplane at the point O, which point O being positioned on thehemispherical surface. A length O₁P₁ and a length OP can be proportionalto each other, satisfying a geometric transformation. A mapping can befrom the point P₁ in the screen to the point P in the tangent plane(rather than from the point P₁ in the screen to a point P_(s) on thehemispherical surface). An angle between O_(r)P and a Z-axis and anangle between O_(r)P and an X-axis can be θ′ and φ′, respectively.

A position of the tangent point O can be determined by the angle of viewparameters θ and p. A radius of the hemisphere and a radius of the imagecan both be 1 as a default.

In process 1), a tangent plane coordinate system X′OY′ can beestablished at the point O. As shown in FIG. 5, the pitch angle θ canvary along an axis OY′ perpendicular to O_(r)O, and the yaw angle φ canvary along an axis OX′ perpendicular to O_(r)O. A dashed box shown onthe semispherical surface can correspond to edges of the screen.

In process 2), the coordinate system O_(r)-XYZ can be transformed to acoordinate system O-X′Y′Z′ through a rotation about the Z-axis by φdegrees, a rotation about the X-axis by θ degrees, and a translationalong the O_(r)O axis from the point O_(r) to the point O. Let acoordinate of the point P in the O-X′Y′Z′ coordinate system be (x′, y′,0) and a coordinate of the point P in the O_(r)-XYZ coordinate system be(x_(p), y_(p), z_(p)), then we have

$\begin{matrix}{\begin{bmatrix}x_{p} \\y_{p} \\z_{p} \\1\end{bmatrix} = {\begin{bmatrix}R & t \\0 & 1\end{bmatrix}\begin{bmatrix}x^{\prime} \\y^{\prime} \\0 \\1\end{bmatrix}}} & (1) \\{\overset{\rightharpoonup}{t} = {{\overset{\rightharpoonup}{O_{r}O}}^{T} = \begin{bmatrix}{\sin\;\theta\;\cos\;\varphi} \\{\sin\;\theta\;\sin\;\varphi} \\{\cos\;\theta}\end{bmatrix}}} & (2) \\{where} & \; \\{R = {\begin{bmatrix}{\sin\;\varphi} & {{- \cos}\;\varphi} & 0 \\{\cos\;\varphi} & {\sin\;\varphi} & 0 \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}1 & 0 & 0 \\0 & {\cos\;\theta} & {{- \sin}\;\theta} \\0 & {\sin\;\theta} & {\cos\;\theta}\end{bmatrix}}} & (3)\end{matrix}$

In process 3), the target image coordinate system xo₁y can betransformed to the tangent plane coordinate system X′OY′. In a casewhere the roll angle ρ=0, there is only a translation between thecoordinate systems xo₁y and X′OY′ without any rotation, and hence:{right arrow over (OP)}=k*{right arrow over (o ₁ p ₁)}  (4)where {right arrow over (o₁p₁)} is a vector in xo₁y, {right arrow over(OP)} is a vector in X′OY′, and k is a scaling factor correlated withthe input parameter γ (angle of view). Referring to FIGS. 5 and 6, wehave the following equation along an OX′ axis:{right arrow over (OA)}=k*{right arrow over (o ₁ a ₁)}  (5)since|O _(r) O|=1  (6)then|OA|=tan(γ/2)  (7)and since|o ₁ a ₁|=1  (8)thenk=tan(γ/2)  (9)

Likewise, the value of the scaling factor k along an OY′ axis can havethe same value.

In a case where there is an axial rotation between the coordinatesystems xo₁y and X′OY′ (for example, the roll angle ρ≠0), the screencoordinates can be rotated, and the equation (4) can be

$\begin{matrix}{\overset{\rightarrow}{OP} = {k*m*\begin{bmatrix}{\cos\;\rho} & {{- \sin}\;\rho} \\{\sin\;\rho} & {\cos\;\rho}\end{bmatrix}\;\overset{\rightarrow}{o_{1}p_{1}}}} & (10)\end{matrix}$where coefficient m guarantees a normalization of the screencoordinates, and m=1/√{square root over (2)} if a width of the screenequals to a height of the screen. Therefore, the following equation canbe obtained:

$\begin{matrix}{\begin{bmatrix}x^{\prime} \\y^{\prime}\end{bmatrix} = {{\tan\left( {\gamma/2} \right)}*{1/\sqrt{2}}*{\begin{bmatrix}{\cos\;\rho} & {{- \sin}\;\rho} \\{\sin\;\rho} & {\cos\;\rho}\end{bmatrix}\begin{bmatrix}x \\y\end{bmatrix}}}} & (11)\end{matrix}$

From coordinates (x, y) of an arbitrary point P₁ in the target image,coordinates (x_(p), y_(p), z_(p)) of a corresponding point P in theworld coordinate system can be calculated by combining equations (1) and(11).

In process 4), the following relationships can be obtained from FIG. 4:θ′=arctan(√{square root over (x _(p) ² +y _(p) ²)}/z _(p))  (12)φ′=arctan(y _(p) /x _(p))  (13)

A coordinate under the world coordinate system can be transformed into acoordinate in the fisheye image.

As shown in FIG. 7, P₂ can be an image point of point P through apinhole imaging. Due to a fisheye distortion, a real image point P′₂ ofpoint P can be positioned in an image before the distortion correction,for example, an input texture-fisheye image. As shown in FIG. 7, O₂ canbe a image point of O in the fisheye image, and can be a central pointof the image.

The correction can be performed using a fisheye lens modelr=f*θ′  (14)where r is a distance from P′₂ to a center O_(f) of the fisheye image,represented as O_(f) P′₂, where f is a parameter of the focal length.Coordinates (u, v) of P′₂ in the fisheye image can be calculated from φ′once r is obtained.

$\begin{matrix}{\begin{bmatrix}u \\v\end{bmatrix} = {r*\begin{bmatrix}{\cos\;\varphi^{\prime}} \\{\sin\;\varphi^{\prime}}\end{bmatrix}}} & (15)\end{matrix}$

A correspondence between coordinates (x, y) of a point P₁ in the targetimage and coordinates (u, v) of the corresponding point P′₂ in thefisheye image can be obtained by combining equations (1), (11), (12),(13), (14) and (15).

A texture information of respective pixel (e.g., a grayscale valuecorresponding to a respective pixel) in the fisheye image can beobtained, such that a texture information can be provided to respectivepoint in the target image. An interpolation can be required in accessingthe pixels as u2, v2 are floating point numbers. The interpolation canbe performed by the GPU.

In this way, a mapping from the screen coordinates P₁ (x, y) to thespherical coordinate system coordinates P (x_(p), y_(p), z_(p)), then tothe fisheye image coordinates P′₂ (u, v), including a coordinate mappingand a texture mapping, can be implemented.

In the example as described hereinabove, a radial distortion isconsidered in the fisheye correction using the fisheye lens model shownin equation (14) where only one distortion correction parameter is used:r=f*θ′  (16)

In order to increase an accuracy of the correction, the above equationcan be modified as:r=Σ _(i=0) ^(n) k _(i)*θ^(2i+1)  (17)where a value of n can be 2. The {k_(i)} can be calibrated in themanufacturing the camera, and provided to the GPU as a uniform variable.The coordinate transformation of each pixel from the fisheye image tothe target image and the distortion correction can be performed by theGPU.

In embodiments of the disclosure, an image processing can be performedbased upon a vibration information. An anti-vibration can be effected inthe process of image processing without using an anti-vibration hardwaredevice (e.g., a gimbal). A cost can be reduced, and a user experiencecan be improved.

FIG. 8 shows a flowchart of a method of image processing in accordancewith another embodiment of the disclosure. The method of imageprocessing in accordance with embodiments of the disclosure can beapplied to various image acquisition devices, and can be implemented byone or more image processors. In some embodiments, the method cancomprise steps S801 to S804.

In step S801, an acquired image captured by an image acquisition devicecan be obtained.

The image acquisition device can include an intelligent image capturingdevice capable of capturing images, such as a still camera or a videocamera. An acquired image can be obtained from the image acquisitiondevice through a data communication therewith. In some instances, theimage acquisition device can comprise a wide-angle lens such as afisheye lens, and the acquired image can be a fisheye image.

In step S802, a vibration information associated with the acquiredimage, which is generated as the image acquisition device capturing theacquired image, can be obtained. In some instances, the vibrationinformation can comprise an angle of vibration.

In some embodiments, the image acquisition device can operate in a videocapture mode or in a continuous capture mode. The vibration informationcan be an angular change in a position of the image acquisition devicein capturing a current image relative to a position of the imageacquisition device in capturing a previous image. Alternatively, thevibration information can be an angular change in a position of theimage acquisition device in capturing a current image relative to areference position. The reference position can be an initial position ofthe image acquisition device where it starts to capture images. In someinstances, the vibration information can include angular changeinformation of the image acquisition device in three directions. Forinstance, the vibration information can comprise angular changeinformation in a pitch angle, a yaw angle and a roll angle.

In some instances, an angular sensor (e.g., a gyroscope) can be providedat the image acquisition device to detect the vibration information ofthe image acquisition device. A raw data can be obtained directly fromthe angular sensor, and the angular change can be calculated from theraw data. Optionally, the angular change can be obtained directly fromthe angular sensor provided that the angular sensor is provided with acalculation capability.

In some instances, the angular sensor can be directly provided on theimage acquisition device through a fixed connection. Alternatively, theangular sensor can be fixedly connected to various external devices towhich the image acquisition device is rigidly connected. For instance,the angular sensor can be mounted on an aerial vehicle which carries theimage acquisition device. The angular sensor fixed connected to theexternal device can detect the vibration information of the imageacquisition device as a vibration of the external devices directly leadsto a vibration of the image acquisition device. It will be appreciatedthat, the angular sensor may not accurately detect the vibrationinformation of the image acquisition device if the image acquisitiondevice is flexibly connected to the external device. In this situation,the angular sensor may not be provided on the external device.

In step S803, a distortion correction can be performed to the acquiredimage based upon the vibration information and a preset distortioncorrection parameter.

In step S804, a target image can be determined from thedistortion-corrected image based upon the vibration information.

In some instances, the target image determined from thedistortion-corrected image can be a portion of the distortion-correctedimage.

In some instances, the step S804 can comprise determining an initialregion from the distortion-corrected image, performing a vibrationcompensation to the initial region based upon the vibration informationto determine a target region, and obtaining the target imagecorresponding to the target region. In some instances, determining theinitial region from the distortion-corrected image can comprisedetermining the initial region from the distortion-corrected image basedupon a preset angle of view.

Optionally, the step S804 can comprise determining an image from thedistortion-corrected image as the target image based upon the vibrationinformation and a preset angle of view, the image having an area definedby a preset angle of view.

In some embodiments of the disclosure, as shown in FIG. 9, adistortion-corrected image 901 can have a first angle of view, while thetarget image can be a partial image cropped from thedistortion-corrected image 901 in accordance with a preset angle of view(e.g., a second angle of view). The partial image can be obtainedthrough a vibration compensation based upon the vibration information,whereas a shaded portion in the distortion-corrected image does notappear in the finally output target image. As shown in FIG. 9, in someembodiments of the disclosure, an image within a dashed box can beobtained if the image acquisition device is rotated downwardly by anangle and no vibration compensation is performed, where the image havingthe angle of view of the target image. On the contrary, a target imagewithin a solid box can be obtained by moving the dashed box upwardly inthe distortion-corrected image if a vibration compensation is performedbased upon the rotated angle.

A mapping from a plane coordinate system in which the acquired image ispositioned to a target coordinate system can be obtained through acoordinate transformation based upon the vibration information, andsubsequently, a texture mapping can be obtained by determining a textureinformation of respect pixel in the acquired image (e.g., a grayscalevalue corresponding to the respective pixel). A conversion from theacquired image to an initial image can be effected based upon thecoordinate mapping, the texture mapping and a preset distortioncorrection parameter, such that a distortion-corrected image can beobtained.

Then, the vibration compensation can be performed based upon thevibration information and the preset angle of view, such that avibration-compensated target image can be obtained from thedistortion-corrected image.

In some instances, the vibration compensation can comprise determiningan initial region in the distortion-corrected image based upon thepreset angle of view, and performing the vibration compensation to theinitial region based upon the vibration information to obtain a targetregion. An image covered by the target region can be the target image.In embodiments of the disclosure, the preset angle of view can besmaller than an angle of view of the image acquisition device incapturing the acquired image. In some instances, the preset angle ofview can be an angle of view corresponding to a standard image, whilethe acquired image can correspond to an angle of view of a fisheyeimage. Optionally, the acquired image can correspond to a standard angleof view, while the preset angle of view can be smaller than the standardangle of view.

For instance, in performing the compensation, if a pitch angleinformation in the vibration information indicates that the imageacquisition device rotates downwardly by a first angle, then the initialregion can be moved upwardly by the first angle. For instance, inperforming the compensation, if a yaw angle information in the vibrationinformation indicates that the image acquisition device rotates to theleft by a second angle, then the initial region can be moved to theright by the second angle. In this way, the vibration compensation canbe effected.

In some instances, the method can further comprise, after the vibrationinformation is detected, detecting an angle of the vibration. The stepS803 can be performed if each one of the pitch angle, yaw angle and rollangle in the angle of vibration is less than a preset threshold angle.Otherwise, the step S801 or S802 can be continuously performed withoutperforming the step S803 if one or more of the pitch angle, yaw angleand roll angle in the angle of vibration exceed the preset thresholdangle, which indicating that a user intentionally moves the imageacquisition device to capture images.

In some instances, the method can further comprise, before performingthe distortion correction to the acquired image based upon the vibrationinformation and the preset distortion correction parameter, performing asynchronization to the obtained acquired image and the vibrationinformation to ensure that the obtained vibration and the obtainedacquired image are associated with each other, and determine the targetimage from the acquired image based upon the vibration information oncethe synchronization is successful.

A detailed description of performing the synchronization to the acquiredimage and the vibration information is provided in embodiments asdiscussed hereinabove with reference to FIGS. 2 to 7.

In some embodiments, the process of performing a distortion correctionto the acquired image based upon the vibration information and thepreset distortion correction parameter can comprises determining amapping from an image coordinate system in which the acquired image ispositioned to a target coordinate system based upon the angle ofvibration, and performing the distortion correction to the acquiredimage to obtain the distortion-corrected image based upon the determinedmapping from the image coordinate system in which the acquired image ispositioned to the target coordinate system and the preset distortioncorrection parameter.

In some embodiments, the process of determining a mapping from the imagecoordinate system in which the acquired image is positioned to a targetcoordinate system based upon the angle of vibration can compriseestablishing a mapping from the image coordinate system in which theacquired image is positioned to a world coordinate system based upon theangle of vibration, determining a mapping from the target coordinatesystem to the world coordinate system based upon the angle of vibration,and determining the mapping from the image coordinate system in whichthe acquired image is positioned to the target coordinate system basedupon the world coordinate system.

In some embodiments, determining a mapping from the target coordinatesystem to the world coordinate system based upon the angle of vibrationcan comprise establishing a mapping from the world coordinate system toa preset tangent plane coordinate system based upon the angle ofvibration, establishing a mapping from the target coordinate system tothe preset tangent plane coordinate system based upon the angle ofvibration, and determining the mapping from the target coordinate systemto the world coordinate system based upon the preset tangent planecoordinate system.

In some instances, the tangent plane coordinate system can be a tangentplane coordinate system in which an arbitrary point on a hemisphericalsurface is positioned, the hemispherical surface having a centercoinciding with a center of a lens of the image acquisition device.

In some embodiments, the process of performing a distortion correctionto the acquired image to obtain the distortion-corrected image basedupon the determined mapping from the image coordinate system in whichthe acquired image is positioned to the target coordinate system and thepreset distortion correction parameter can comprise: mapping each pointin the acquired image to a corresponding location in the targetcoordinate system based upon the determined mapping from the imagecoordinate system in which the acquired image is positioned to thetarget coordinate system and the distortion correction parameter,obtaining a pixel value of each point in the acquired image, andgenerating the distortion-corrected image based upon the obtained pixelvalue and the location of each point.

A detailed description of the coordinate mapping and the distortioncorrection is provided in embodiments as discussed hereinabove withreference to FIGS. 2 to 7.

In embodiments of the disclosure, an image processing can be performedbased upon a vibration information. An anti-vibration can be effected inthe process of image processing without using an anti-vibration hardwaredevice (e.g., a gimbal). A cost can be reduced, and a user experiencecan be improved.

An device of image processing and a camera in accordance withembodiments of the disclosure will be described.

FIG. 10 shows a configuration of an device of image processing inaccordance with embodiments of the disclosure. The device of imageprocessing in accordance with embodiments of the disclosure can beprovided in various cameras. Optionally, the device of image processingcan be used as a standalone device. In some embodiments, the device ofimage processing can comprise an image obtaining module 11, a vibrationobtaining module 12 and a processing module 13. The image obtainingmodule 11 can be configured to obtain an acquired image captured by animage acquisition device. The vibration obtaining module 12 can beconfigured to obtain a vibration information associated with theacquired image, which is generated as the image acquisition devicecapturing the acquired image. In some instances, the vibrationinformation can comprise an angle of vibration. The processing module 13can be configured to determine a target image from the acquired imagebased upon the vibration information.

The image acquisition device can be any intelligent image capturingdevice capable of capturing images (e.g., a still camera or a videocamera) in data communication with the image processing device. Theimage obtaining module 11 can obtain the acquired image from the imageacquisition device once in data communication with the image acquisitiondevice. In some instances, the image acquisition device can comprise awide-angle lens such as a fisheye lens, and the acquired image acquiredcan be a fisheye image.

In some embodiments, the vibration information obtained by the vibrationobtaining module 12 can be an angular change in a position of the imageacquisition device in capturing the acquired image relative to aposition of the image acquisition device in capturing a previouslyacquired image. The angle of vibration can comprise a pitch angle, a yawangle and/or a roll angle. Alternatively, the vibration information canbe an angular change in a position of the image acquisition device incapturing the acquired image relative to a reference position.

In some embodiments, the image acquisition device can operate in a videocapture mode or in a continuous capture mode. The vibration informationcan be an angular change in a position of the image acquisition devicein capturing a current image relative to a position of the imageacquisition device in capturing a previous image. Alternatively, thevibration information can be an angular change in a position of theimage acquisition device in capturing a current image relative to areference position. The reference position can be an initial position ofthe image acquisition device where it starts to capture images. In someinstances, the vibration information can include an angle information ofthe image acquisition device in three directions. For instance, thevibration information can comprise a pitch angle, a yaw angle and a rollangle.

In some instances, the vibration obtaining module 12 can obtain thevibration information of the image acquisition device using an angularsensor (e.g., a gyroscope) provided in the image acquisition device. Thevibration obtaining module 12 can obtain a raw data directly from theangular sensor, and subsequently calculate the angle information.Optionally, the vibration obtaining module 12 can obtain the angleinformation directly from the angular sensor provided that the angularsensor is provided with a calculation capability.

In some embodiments, the angular sensor can be directly provided on theimage acquisition device through a fixed connection. Alternatively, theangular sensor can be fixedly connected to various external devices towhich the image acquisition device is rigidly connected. For instance,the angular sensor can be mounted on an aerial vehicle which carries theimage acquisition device. The angular sensor fixed connected to theexternal device can detect the vibration information of the imageacquisition device as a vibration of the external devices directly leadsto a vibration of the image acquisition device. It will be appreciatedthat, the angular sensor may not accurately detect the vibrationinformation of the image acquisition device if the image acquisitiondevice is flexibly connected to the external device. In this situation,the angular sensor may not be provided on the external device.

In some instances, the target image can be an image obtained by theprocessing module 13 through a coordinate transformation and adistortion correction based upon the vibration information. Optionally,the target image can be a partial image cropped from an image based uponthe vibration information, which image is processed through a coordinatetransformation and a distortion correction.

In some embodiments, the device in accordance with embodiments of thedisclosure can further comprise a correction module 14. The correctionmodule 14 can be configured to perform a synchronization to the obtainedacquired image and the vibration information, and provide a triggerinformation to the processing module once the synchronization issuccessful.

The acquired image and the vibration information can be aligned witheach other in a timeline through the time correction performed by thecorrection module 14 to the data obtained by the image obtaining module11 and the vibration obtaining module 12.

In some embodiments, as shown in FIG. 11, the processing module 13 cancomprise a mapping determining unit 131 and a processing unit 132. Themapping determining unit 131 can be configured to determine a mappingfrom an image coordinate system in which the acquired image ispositioned to a target coordinate system based upon the angle ofvibration. The processing unit 132 can be configured to determine thetarget image from the acquired image based upon the determined mappingfrom the image coordinate system in which the acquired image ispositioned to the target coordinate system.

In some embodiments, the mapping determining unit 131 can comprise afirst determining subunit 1311, a second determining subunit 1312 and athird determining subunit 1313. The first determining subunit 1311 canbe configured to establish a mapping from the image coordinate system inwhich the acquired image is positioned to a world coordinate systembased upon the angle of vibration. The second determining subunit 1312can be configured to determine a mapping from the target coordinatesystem to the world coordinate system based upon the angle of vibration.The third determining subunit 1313 can be configured to determine themapping from the image coordinate system in which the acquired image ispositioned to the target coordinate system based upon the worldcoordinate system.

In some embodiments, the second determining subunit 1312 can beconfigured to establish a mapping from the world coordinate system to apreset tangent plane coordinate system based upon the angle ofvibration, establish a mapping from the target coordinate system to thepreset tangent plane coordinate system based upon the angle ofvibration, and determine the mapping from the target coordinate systemto the world coordinate system based upon the preset tangent planecoordinate system.

In some instances, the tangent plane coordinate system can be a tangentplane coordinate system in which an arbitrary point on a hemisphericalsurface is positioned, the hemispherical surface having a centercoinciding with a center of a lens of the image acquisition device.

In some embodiments, the processing unit 132 can be configured to obtaina preset distortion correction parameter, map each point in the acquiredimage to a corresponding location in the target coordinate system basedupon the determined mapping from the image coordinate system in whichthe acquired image is positioned to the target coordinate system and thedistortion correction parameter, obtain a pixel value of each point inthe acquired image, and generate the target image based upon theobtained pixel value and the location of each point.

In some embodiments, the device of image processing in accordance withembodiments of the disclosure can be provided in a processor. Theprocessor can be a part of an aerial vehicle for aerial photography. Theaerial vehicle for aerial photography can comprise a body, the imageacquisition device and the processor which includes the device of imageprocessing in accordance with embodiments of the disclosure. The imageacquisition device can be mounted on the body of the aerial vehicle. Theprocessor can be in data communication with the image acquisitiondevice.

A detailed description of the modules, units and subunits in the deviceof image processing is provided in method embodiments as discussedhereinabove with reference to FIGS. 1 to 8.

In embodiments of the disclosure, an image processing can be performedbased upon a vibration information. An anti-vibration can be effected inthe process of image processing without using an anti-vibration hardwaredevice (e.g., a gimbal). A cost can be reduced, and a user experiencecan be improved.

FIG. 12 shows a configuration of a camera as an example of the imageacquisition device in accordance with embodiments of the disclosure. Thecamera in accordance with embodiments of the disclosure can comprise alens 01, an image sensor 02 and an image processor 03. The image sensor02 can be configured to acquire an image data through the lens 01. Theimage processor 03 can be connected to the image sensor 02 andconfigured to obtain an acquired image from the image data acquired bythe image sensor 02, obtain a vibration information associated with theacquired image, which vibration information is generated as the imageacquisition device capturing the acquired image, and determine a targetimage from the acquired image based upon the vibration information. Insome instances, the vibration information can comprise an angle ofvibration.

In some instances, the vibration information can be an angular change ina position of the image acquisition device in capturing the acquiredimage relative to a position of the image acquisition device incapturing a previously acquired image. The angle of vibration cancomprise a pitch angle, a yaw angle and/or a roll angle. Optionally, thevibration information can be an angular change in a position of theimage acquisition device in capturing the acquired image relative to areference position.

In some embodiments, the image processor 03 can be configured to performa synchronization to the obtained captured image and the vibrationinformation, and determine the target image from the acquired imagebased upon the vibration information once the synchronization issuccessful.

In some embodiments, the lens 01 can be a fisheye lens 01, and theacquired image can be a fisheye image.

In some embodiments, the image processor 03 can be configured todetermine a mapping from an image coordinate system in which theacquired image is positioned to a target coordinate system based uponthe angle of vibration, and determine the target image from the acquiredimage based upon the determined mapping from the image coordinate systemin which the acquired image is positioned to the target coordinatesystem.

In some embodiments, the image processor 03 can be configured toestablish a mapping from the image coordinate system in which theacquired image is positioned to a world coordinate system based upon theangle of vibration, determine a mapping from the target coordinatesystem to the world coordinate system based upon the angle of vibration,and determine a mapping from the image coordinate system in which theacquired image is positioned to the target coordinate system based uponthe world coordinate system.

In some embodiments, the image processor 03 can be configured toestablish a mapping from the world coordinate system to a preset tangentplane coordinate system based upon the angle of vibration, establish amapping from the target coordinate system to the preset tangent planecoordinate system based upon the angle of vibration, and determine amapping from the target coordinate system to the world coordinate systembased upon the preset tangent plane coordinate system.

In some instances, the tangent plane coordinate system can be a tangentplane coordinate system in which an arbitrary point on a hemisphericalsurface having a center coinciding with a center of a lens of the imageacquisition device is positioned.

In some embodiments, the image processor 03 can be configured to obtaina preset distortion correction parameter, map each point in the acquiredimage to a corresponding location in the target coordinate system basedupon the determined mapping from the image coordinate system in whichthe acquired image is positioned to the target coordinate system and thedistortion correction parameter, obtain a pixel value of each point inthe acquired image, and generate the target image based upon theobtained pixel value and the location of each point.

The camera in accordance with embodiments of the disclosure can be apart of an aerial vehicle for aerial photography. The aerial vehicle foraerial photography can comprise a body and the camera. The camera can bemounted on the body of the aerial vehicle.

A detailed description of components of the camera is provided in methodembodiments as discussed hereinabove with reference to FIGS. 1 to 8.

In embodiments of the disclosure, an image processing can be performedbased upon a vibration information. An anti-vibration can be effected inthe process of image processing without using an anti-vibration hardwaredevice (e.g., a gimbal). A cost can be reduced, and a user experiencecan be improved.

FIG. 13 shows a configuration of a device of image processing inaccordance with another embodiment of the disclosure. In some instances,the device of image processing in accordance with the embodiments of thedisclosure can be provided in various cameras. Optionally, the device ofimage processing can be used as a standalone device. The device inaccordance with embodiments of the disclosure can comprise a firstobtaining module 21, a second obtaining module 22, a first processingmodule 23 and a second processing module 24.

The first obtaining module 21 can be configured to obtain an acquiredimage captured by an image acquisition device. The second obtainingmodule 22 can be configured to obtain a vibration information associatedwith the acquired image, which vibration information is generated as theimage acquisition device capturing the acquired image. In someinstances, the vibration information can comprise an angle of vibration.The first processing module 23 can be configured to perform a distortioncorrection to the acquired image based upon the vibration informationand a preset distortion correction parameter. The second processingmodule 24 can be configured to determine a target image from thedistortion-corrected image based upon the vibration information.

In some embodiments, the target image determined by the secondprocessing module 24 can be a part of the distortion-corrected image.

In some embodiments, the second processing module 24 can be configuredto determine an initial region from the distortion-corrected image,perform a vibration compensation to the initial region based upon thevibration information to determine a target region, and obtain thetarget image corresponding to the target region.

In some embodiments, the second processing module 24 can be configuredto determine the initial region from the distortion-corrected imagebased upon a preset angle of view.

In some embodiments, the second processing module 24 can be configuredto determine an image from the distortion-corrected image as the targetimage based upon the vibration information and the preset angle of view,the image having an area defined by the preset angle of view.

The image acquisition device can be any intelligent image capturingdevice capable of capturing images (e.g., a still camera or a videocamera). The first obtaining module 21 can obtain the acquired imagefrom the image acquisition device through a data communication with theexternal image acquisition device. In some instances, the imageacquisition device can comprise a wide-angle lens, such as a fisheyelens, and the acquired image can be a fisheye image.

In some embodiments, the image acquisition device can operate in a videocapture mode or in a continuous capture mode. The vibration informationobtained by the second obtaining module 22 can be an angular change in aposition of the image acquisition device in capturing a current imagerelative to a position of the image acquisition device in capturing aprevious image. Alternatively, the vibration information can be anangular change in a position of the image acquisition device incapturing a current image relative to a reference position. Thereference position can be an initial position of the image acquisitiondevice where it starts to capture images. In some instances, thevibration information can include an angle information of the imageacquisition device in three directions. For instance, the vibrationinformation can comprise a pitch angle, a yaw angle and a roll angle.

In some embodiments, the second obtaining module 22 can obtain thevibration information of the image acquisition device through an angularsensor (e.g., a gyroscope) provided in the image acquisition device. Thesecond obtaining module 22 can obtain a raw data directly from theangular sensor, and subsequently calculate the angle information.Optionally, the second obtaining module 22 can obtain the angleinformation directly from the angular sensor provided that the angularsensor is provided with a calculation capability.

In some embodiments, the angular sensor can be directly provided on theimage acquisition device through a fixed connection. Alternatively, theangular sensor can be fixedly connected to various external devices towhich the image acquisition device is rigidly connected. For instance,the angular sensor can be mounted on an aerial vehicle which carries theimage acquisition device. The angular sensor fixed connected to theexternal device can detect the vibration information of the imageacquisition device as a vibration of the external devices directly leadsto a vibration of the image acquisition device. It will be appreciatedthat, the angular sensor may not accurately detect the vibrationinformation of the image acquisition device if the image acquisitiondevice is flexibly connected to the external device. In this situation,the angular sensor may not be provided on the external device.

In some embodiments, the second processing module 24 can be configuredto determine the initial region from the distortion-corrected imagebased upon the preset angle of view.

The first processing module 23 can obtain a mapping from a planecoordinate system in which the acquired image is positioned to a targetcoordinate system through a coordinate transformation based upon thevibration information, and subsequently obtain a texture mapping bydetermining a texture information of respective pixel in the acquiredimage (e.g., a grayscale value corresponding to the respective pixel). Aconversion from the acquired image to an initial image can be effectedbased upon the coordinate mapping, the texture mapping and a presetdistortion correction parameter, such that a distortion-corrected imagecan be obtained.

The second processing module 24 can then perform the vibrationcompensation based upon the vibration information and the preset angleof view, such that a vibration-compensated target image can be obtainedfrom the distortion-corrected image.

In some instances, the vibration compensation can comprise determiningan initial region in the distortion-corrected image based upon thepreset angle of view, and performing the vibration compensation to theinitial region based upon the vibration information to obtain a targetregion. An image covered by the target region can be the target image.In embodiments of the disclosure, the preset angle of view can besmaller than an angle of view of the image acquisition device incapturing the acquired image. In some instances, the preset angle ofview can be an angle of view corresponding to a standard image, whilethe acquired image can correspond to an angle of view of a fisheyeimage. Optionally, the acquired image can correspond to a standard angleof view, while the preset angle of view can be smaller than the standardangle of view.

For instances, in performing the compensation, if a pitch angleinformation in the vibration information indicates that the imageacquisition device rotates downwardly by a first angle, then the secondprocessing module 24 can reversely move the initial region upwardly bythe first angle. For instance, if a yaw angle information in thevibration information indicates that the image acquisition devicerotates to the left by a second angle, then the second processing module24 can reversely move the initial region to the right by the secondangle. In this way, the vibration compensation can be effected.

In some embodiments, the device can further comprise a detection judgingmodule configured to detect an angle of the vibration after thevibration information is detected. The detection judging module can beconfigured to send a trigger information to the first processing module23 to perform the distortion correction if each one of the pitch angle,yaw angle and roll angle in the angle of vibration is less than a presetthreshold angle. Otherwise, if one or more of the pitch angle, yaw angleand roll angle in the angle of vibration exceed the preset thresholdangle, which indicating that a user intentionally moves the imageacquisition device to capture images, the distortion correction and theprocessing performed to the target image may not be required andsubsequent operations of the first processing module 23 and the secondprocessing module 24 can be stopped, or the detection judging module cansend a trigger information to the first obtaining module 21 and thesecond obtaining module 22 to obtain relevant data.

In some embodiments, the second processing module 24 can be configuredto determine an initial region from the distortion-corrected image basedupon the preset angle of view.

In some instances, the vibration information can be an angular change ina position of the image acquisition device in capturing the acquiredimage relative to a position of the image acquisition device incapturing a previously acquired image. The angle of vibration cancomprise a pitch angle, a yaw angle and/or a roll angle.

In some embodiments, the device of image processing in accordance withembodiments of the disclosure can further comprise a correction module25. The correction module 25 can be configured to perform asynchronization to the obtained acquired image and the vibrationinformation, and send a trigger information to the first processingmodule 23 once the synchronization is successful.

The correction module 25 can perform a correction to the data obtainedby the first obtaining module 21 and the second obtaining module 22,such that the acquired image and the vibration information can bealigned with each other in a timeline, ensuring that the vibrationinformation is an information on the angular change in the position ofthe image acquisition device at a time the acquired image is captured.

In some embodiments, as shown in FIG. 14, the first processing module 23can comprise a mapping determining unit 231 and an image determiningunit 232. The mapping determining unit 231 can be configured todetermine a mapping from an image coordinate system in which theacquired image is positioned to a target coordinate system based uponthe angle of vibration. The image determining unit 232 can be configuredto perform the distortion correction to the acquired image to obtain thedistortion-corrected image based upon the determined mapping from theimage coordinate system in which the acquired image is positioned to thetarget coordinate system and the preset distortion correction parameter.

In some embodiments, the mapping determining unit 231 can comprise afirst determining subunit 2311, a second determining subunit 2312 and athird determining subunit 2313. The first determining subunit 2311 canbe configured to establish a mapping from the image coordinate system inwhich the acquired image is positioned to a world coordinate systembased upon the angle of vibration. The second determining subunit 2312can be configured to determine a mapping from the target coordinatesystem to the world coordinate system based upon the angle of vibration.The third determining subunit 2313 can be configured to determine amapping from the image coordinate system in which the acquired image ispositioned to the target coordinate system based upon the worldcoordinate system.

In some embodiments, the second determining subunit 2312 can beconfigured to establish a mapping from the world coordinate system to apreset tangent plane coordinate system based upon the angle ofvibration, establish a mapping from the target coordinate system to thepreset tangent plane coordinate system based upon the angle ofvibration, and determine a mapping from the target coordinate system tothe world coordinate system based upon the preset tangent planecoordinate system.

In some instances, the tangent plane coordinate system can be a tangentplane coordinate system in which an arbitrary point on a hemisphericalsurface is positioned, the hemispherical surface having a centercoinciding with a center of a lens of the image acquisition device.

In some embodiments, the image determining unit 232 can be configured tomap each point in the acquired image to a corresponding location in thetarget coordinate system based upon the determined mapping from theimage coordinate system in which the acquired image is positioned to thetarget coordinate system and the distortion correction parameter, obtaina pixel value of each point in the acquired image, and generate thedistortion-corrected image based upon the obtained pixel value and thelocation of each point.

The device of image processing in accordance with embodiments of thedisclosure can be provided in a processor. The processor can be a partof an aerial vehicle for aerial photography. The aerial vehicle foraerial photography can comprise a body, the image acquisition device andthe processor which includes the device of image processing inaccordance with embodiments of the disclosure. The image acquisitiondevice can be mounted on the body of the aerial vehicle. The processorcan be in data communication with the image acquisition device.

A detailed description of the modules, units and subunits in the deviceof image processing is provided in method embodiments as discussedhereinabove with reference to FIGS. 1 to 8.

In embodiments of the disclosure, an image processing can be performedbased upon a vibration information. An anti-vibration can be effected inthe process of image processing without using an anti-vibration hardwaredevice (e.g., a gimbal). A cost can be reduced, and a user experiencecan be improved.

FIG. 15 shows a configuration of a camera as an example imageacquisition device in accordance with embodiments of the disclosure. Thecamera in accordance with embodiments of the disclosure can comprise alens 04, an image sensor 05 and an image processor 06. The image sensor05 can be configured to acquire an image data through the lens 04. Theimage processor 06 can be connected to the image sensor 05. The imageprocessor 06 can be configured to: obtain an acquired image from theimage data acquired by the image sensor 05; obtain a vibrationinformation associated with the acquired image, which vibrationinformation is generated as the image acquisition device capturing theacquired image, which vibration information comprising an angle ofvibration; perform a distortion correction to the acquired image basedupon the vibration information and a preset distortion correctionparameter; and determine a target image from the distortion-correctedimage based upon the vibration information.

In some instances, the target image determine by the image processor 06can be a part of the distortion-corrected image.

In some instances, the image processor 06 can be configured to determinean initial region from the distortion-corrected image, perform avibration compensation to the initial region based upon the vibrationinformation to determine a target region, and obtain the target imagecorresponding to the target region.

In some instances, the image processor 06 can be configured to determinethe initial region from the distortion-corrected image based upon apreset angle of view.

In some instances, the image processor 06 can be configured to determinean image from the distortion-corrected image as the target image basedupon the vibration information and the preset angle of view, the imagehaving an area defined by the preset angle of view.

In some instances, the image processor 06 can be configured to determinethe initial region from the distortion-corrected image based upon thepreset angle of view.

In some instances, the vibration information can be an angular change ina position of the image acquisition device in capturing the acquiredimage relative to a position of the image acquisition device incapturing a previously acquired image. Optionally, the vibrationinformation can be an angular change in a position of the imageacquisition device in capturing the acquired image relative to areference position. The angle of vibration can comprise a pitch angle, ayaw angle and/or a roll angle.

In some instances, the image processor 06 can be further configured toperform a synchronization to the obtained acquired image and thevibration information, and determine the target image from the acquiredimage based upon the vibration information once the synchronization issuccessful.

In some instances, the lens 04 can be a fisheye lens, and the acquiredimage can be a fisheye image.

In some instances, the image processor 06 can be configured to determinea mapping from an image coordinate system in which the acquired image ispositioned to a target coordinate system based upon the angle ofvibration, and perform the distortion correction to the acquired imageto obtain the distortion-corrected image based upon the determinedmapping from the image coordinate system in which the acquired image ispositioned to the target coordinate system and the preset distortioncorrection parameter.

In some instances, the image processor 06 can be configured to establisha mapping from the image coordinate system in which the acquired imageis positioned to a world coordinate system based upon the angle ofvibration, determine a mapping from the target coordinate system to theworld coordinate system based upon the angle of vibration, and determinea mapping from the image coordinate system in which the acquired imageis positioned to the target coordinate system based upon the worldcoordinate system.

In some instances, the image processor 06 is configured to establish amapping from the world coordinate system to a preset tangent planecoordinate system based upon the angle of vibration, establish a mappingfrom the target coordinate system to the preset tangent plane coordinatesystem based upon the angle of vibration, and determine a mapping fromthe target coordinate system to the world coordinate system based uponthe preset tangent plane coordinate system.

In some instances, the tangent plane coordinate system can be a tangentplane coordinate system in which an arbitrary point on a hemisphericalsurface is positioned, the hemispherical surface having a centercoinciding with a center of a lens of the image acquisition device.

In some instances, the image processor 06 can be configured to map eachpoint in the acquired image to a corresponding location in the targetcoordinate system based upon the determined mapping from the imagecoordinate system in which the acquired image is positioned to thetarget coordinate system and the distortion correction parameter, obtaina pixel value of each point in the acquired image, and generate thedistortion-corrected image based upon the obtained pixel value and thelocation of each point.

The camera in accordance with embodiments of the disclosure can be apart of an aerial vehicle for aerial photography. The aerial vehicle foraerial photography can comprise a body and the camera. The camera can bemounted on the body of the aerial vehicle.

A detailed description of the components of the camera is provided inmethod embodiments as discussed hereinabove with reference to FIGS. 1 to8.

In embodiments of the disclosure, an image processing can be performedbased upon a vibration information. An anti-vibration can be effected inthe process of image processing without using an anti-vibration hardwaredevice (e.g., a gimbal). A cost can be reduced, and a user experiencecan be improved.

It will be appreciated that, the device and method disclosed inembodiments of the disclosure can be implemented in other manners. Forinstance, the described device embodiments are merely illustrative. Forinstance, a division of modules or units is merely a division based upona logical function. Various divisions can be possible in actualimplementation. For instance, multiple units or components can becombined or integrated on another system. For instance, some featurescan be ignored or not be performed. For instance, a mutual coupling, adirect coupling or a communication connection as shown or discussed canbe an indirect coupling or a communication connection via an interface,a means or a unit. The coupling can be an electrical coupling or amechanical coupling.

The units illustrated as separate parts may or may not be physicallyseparated. The parts shown as units may or may not be physical units.For instance, the parts can be provided at the same location ordistributed over a plurality of network units. All or part of the unitscan be selected to implement the embodiments of the disclosure accordingto actual requirements.

Various functional units in the embodiments of the disclosure may beintegrated in one processing unit. The functional units can be separateand physical units. Two or more units may be integrated in one unit. Theintegrated units may be implemented as hardware or software functionalunits.

The integrated units, if implemented as software functional units andsold or used as independent products, may be stored in acomputer-readable storage medium. With such an understanding,essentially the technical solution of the disclosure, or a part makingcontribution over the prior art, or all or part of the technicalsolution may be embodied as a software product. The computer softwareproduct is stored in a storage medium and includes several instructionsfor causing a computer processor to execute all or part of steps of themethod according to the various embodiments of the present disclosure.The above mentioned storage medium includes: various media capable ofstoring program code, such as a U disk, a removable hard disk, ROM(read-only memory), RAM (random access memory), a diskette, an opticaldisk, etc.

The above description merely illustrates some embodiments of thedisclosure and is not intended to limit the scope of the disclosure. Anyequivalent changes in structures or processes made in light of thespecification and the drawings, and their direct or indirect applicationin other related technical fields should all be encompassed in the scopeof the present disclosure.

What is claimed is:
 1. An image processing method comprising: obtainingan acquired image captured by an image acquisition device; obtainingvibration information associated with the acquired image and generatedwhile the image acquisition device capturing the acquired image, thevibration information comprising an angular change; determining amapping from an image coordinate system in which the acquired image ispositioned to a target coordinate system based upon the angular change;performing a distortion correction to the acquired image to obtain thedistortion-corrected image based upon the mapping from the imagecoordinate system to the target coordinate system and the presetdistortion correction parameter; and determining a target image from thedistortion-corrected image based upon the vibration information; whereindetermining the mapping from the image coordinate system to the targetcoordinate system based upon the angular change comprises: establishinga mapping from the image coordinate system to a world coordinate systembased upon the angular change; determining a mapping from the targetcoordinate system to the world coordinate system based upon the angularchange; and determining the mapping from the image coordinate system tothe target coordinate system based upon the world coordinate system;wherein determining the mapping from the target coordinate system to theworld coordinate system based upon the angular change comprises:establishing a mapping from the world coordinate system to a presettangent plane coordinate system based upon the angular change;establishing a mapping from the target coordinate system to the presettangent plane coordinate system based upon the angular change; anddetermining the mapping from the target coordinate system to the worldcoordinate system based upon the preset tangent plane coordinate system.2. The method of claim 1, wherein determining the target image from thedistortion-corrected image based upon the vibration informationcomprises: determining an initial region from the distortion-correctedimage, performing a vibration compensation to the initial region basedupon the vibration information to determine a target region, andobtaining the target image corresponding to the target region.
 3. Themethod of claim 2, wherein determining the initial region from thedistortion-corrected image comprises: determining the initial regionfrom the distortion-corrected image based upon a preset angle of view.4. The method of claim 1, wherein determining the target image from thedistortion-corrected image based upon the vibration informationcomprises: determining an image from the distortion-corrected image asthe target image based upon the vibration information and a preset angleof view, the image corresponding to an area defined by the angle ofview.
 5. The method of claim 1, wherein: the angular change includes achange of the position of the image acquisition device in capturing theacquired image relative to the position of the image acquisition devicein capturing a previously acquired image or relative to a referenceposition, and the angular change comprises at least one of a pitchangle, a yaw angle, or a roll angle.
 6. The method of claim 1, furthercomprising, before performing the distortion correction to the acquiredimage based upon the vibration information and the preset distortioncorrection parameter: performing a synchronization to the acquired imageand the vibration information.
 7. The method of claim 1, wherein theacquired image includes a fisheye image.
 8. The method of claim 1,wherein performing the distortion correction to the acquired image toobtain the distortion-corrected image based upon the mapping from theimage coordinate system to the target coordinate system and the presetdistortion correction parameter comprises: mapping a plurality of pointsin the acquired image to corresponding locations in the targetcoordinate system based upon the determined mapping from the imagecoordinate system to the target coordinate system and the distortioncorrection parameter; obtaining pixel values of the points in theacquired image; and generating the distortion-corrected image based uponthe pixel values and the locations of the points in the targetcoordinate system.
 9. An image processing device comprising: aprocessor; and a memory storing instructions that, when executed by theprocessor, cause the processor to: obtain an acquired image captured byan image acquisition device; obtain vibration information associatedwith the acquired image and generated while the image acquisition devicecapturing the acquired image, the vibration information comprising anangular change; establish a mapping from an image coordinate system inwhich the acquired image is positioned to a world coordinate systembased upon the angular change; determine a mapping from a targetcoordinate system to the world coordinate system based upon the angularchange; establish a mapping from the world coordinate system to a presettangent plane coordinate system based upon the angular change; establisha mapping from the target coordinate system to the preset tangent planecoordinate system based upon the angular change; determine a mappingfrom the target coordinate system to the world coordinate system basedupon the preset tangent plane coordinate system; determine a mappingfrom the image coordinate system to the target coordinate system basedupon the world coordinate system; perform a distortion correction to theacquired image to obtain the distortion-corrected image based upon themapping from the image coordinate system to the target coordinate systemand the preset distortion correction parameter; and determine a targetimage from the distortion-corrected image based upon the vibrationinformation.
 10. The device of claim 9, wherein the instructions furthercause the processor to: determine an initial region from thedistortion-corrected image; perform a vibration compensation to theinitial region based upon the vibration information to determine atarget region; and obtain the target image corresponding to the targetregion.
 11. The device of claim 10, wherein the instructions furthercause the processor to: determine the initial region from thedistortion-corrected image based upon a preset angle of view.
 12. Thedevice of claim 9, wherein the instructions further cause the processorto: determine an image from the distortion-corrected image as the targetimage based upon the vibration information and a preset angle of view,the image corresponding to an area defined by the angle of view.
 13. Thedevice of claim 9, wherein: the angular change includes a change of theposition of the image acquisition device in capturing the acquired imagerelative to the position of the image acquisition device in capturing apreviously acquired image or relative to a reference position, and theangular change comprises at least one of a pitch angle, a yaw angle, ora roll angle.
 14. The device of claim 9, wherein the instructionsfurther cause the processor to, before performing the distortioncorrection to the acquired image based upon the vibration informationand the preset distortion correction parameter: perform asynchronization to the acquired image and the vibration information. 15.The device of claim 9, wherein the instructions further cause theprocessor to: map a plurality of points in the acquired image tocorresponding locations in the target coordinate system based upon thedetermined mapping from the image coordinate system to the targetcoordinate system and the distortion correction parameter; obtain pixelvalues of the points in the acquired image; and generate thedistortion-corrected image based upon the pixel values and the locationsof the points in the target coordinate system.
 16. An aerial vehiclecomprising: a body; an image acquisition device mounted on the body; andan image processing device of claim 9 in data communication with theimage acquisition device.